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3.  Long  Term  Goals  of  the  Project 

3.1  To  develop  a  theory  of  the  cognitive  processes  involved  in  understanding  diagrams 
and  other  visualizations  of  causal  systems  and  making  inferences  from  these 
visualizations 
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3.2  To  translate  this  knowledge  into  practical  guidelines  for  designing  visualizations  of 
causal  phenomena  and  to  validate  the  guidelines  by  designing  and  testing  prototypes. 

4.  Scientific  and  Technical  Objectives 

4.1  To  understand  how  novices  and  experts  comprehend  complex  visualizations  in 
domains  such  as  mechanics  and  meteorology  and  how  they  make  causal  inferences  from 
these  visualizations. 

4.2  To  develop  design  principles  for  interactive  visualizations  in  these  complex  domains 
and  to  build  and  evaluate  prototypes  embodying  these  principles. 

5.  Approach 

5.1  Developing  a  model  of  how  experts  and  novices  comprehend  and  make  inferences 
from  complex  visualizations 

5.2  Testing  and  developing  this  model  by  conducting  empirical  studies  of  how  experts 
and  novices  comprehend  and  make  inferences  from  complex  visualizations.  These  studies 
involve  collection  and  analysis  of  several  different  types  of  data,  including  problem 
solving  accuracy,  response  times,  eye-fixation  protocols,  and  verbal  protocols. 

5.3  Development  of  design  principles  for  interactive  visualizations  that  support  human 
performance,  based  on  the  model  and  empirical  studies  described  above. 

6.  Concise  Progress  Summary 

6.1  A  Preliminary  Model  of  Comprehension  and  Inference  from  Complex 
Visualizations 

In  research  conducted  with  funding  from  the  Office  of  Naval  Research  in  the  last  6 
years,  we  have  developed  a  model  of  how  people  understand  and  make  inferences  from 
multimedia  presentations  (Narayanan  &  Hegarty,  1998;  Hegarty,  Quilici,  Narayanan, 
Holmquist  and  Moreno,  1999).  This  model  was  initially  developed  in  the  domain  of 
mechanics  and  has  been  generalized  to  the  domains  of  computer  algorithms  (Hansen, 
Narayanan  &  Hegarty,  2002)  and  meteorology  in  more  recent  research. 

According  to  this  model,  people  construct  a  mental  model  of  a  dynamic  system  (e.g. 
a  machine)  by  first  decomposing  it  into  simpler  components,  retrieving  relevant 
background  knowledge  about  these  components,  and  mentally  encoding  the  relations 
(spatial  and  semantic)  between  components  to  construct  a  static  mental  model.  They  then 
mentally  animate  this  static  mental  model  to  construct  a  kinematic/dynamic  mental  model 
of  the  system.  We  postulate  that  mental  model  construction  under  these  circumstances 


requires  the  stages  described  below.  Although  we  list  them  in  order,  we  acknowledge _ 

that  they  are  not  always  accomplished  in  this  order. 

A.  Machine  Decomposition  by  Diagram  Parsing.  Diagrams  of  mechanical  systems 
are  made  up  of  elementary  shapes,  such  as  rectangles,  circles  and  cylinders,  which 
represent  objects  such  as  pistons,  gears  and  tubes.  The  first  step  in  comprehension  is  to 
parse  the  connected  diagram  into  these  elementary  shapes,  i.e.,  units  in  the  diagram  that 
correspond  to  subcomponents  of  the  mechanical  system. 

B.  Constructing  a  Static  Mental  Model  by  Making  Representational  Connections. 

The  second  stage  in  multi-modal  comprehension  involves  making  appropriate 
connections  in  memory  among  the  components  identified  in  Stage  1.  This  stage  involves 
making  two  types  of  connections:  (1)  connections  to  prior  knowledge  and  (2)  connections 
to  the  representations  of  other  machine  components. 

First,  the  user  must  identify  the  components,  that  is,  make  connections  between  the 
diagrammatic  elements  identified  at  Stage  1  and  their  real-world  referents.  For  example, 
the  user  might  represent  that  a  rectangle  represents  a  piston  or  a  circle  represents  a  gear. 
Prior  knowledge  can  also  provide  additional  information  about  components,  such  as  what 
they  are  typically  made  of  and  whether  they  are  rigid  or  flexible.  This  information  is 
valuable  in  making  inferences  about  how  components  move  and  constrain  each  other’s 
behaviors. 

Second,  the  user  must  represent  the  spatial  relations  between  different  machine 
components  by  building  connections  between  the  representations  of  these  components 
(Mayer  &  Sims,  1994).  In  understanding  how  a  machine  works,  information  about  the 
spatial  relations  between  mechanical  components  forms  the  basis  for  inferences  about  the 
motions  of  components,  because  these  spatial  relations  determine  how  components  affect 
and  constrain  each  other’s  motions. 

C.  Making  Referential  Connections.  When  diagrams  are  accompanied  by  text,  an 
additional  stage  in  comprehension  is  that  of  resolving  coreference  between  the  two 
media,  i.e.,  making  referential  links  between  a  noun  phrase  in  the  text  (e.g.,  "the  piston") 
and  the  diagrammatic  unit  that  depicts  its  referent  (e.g.,  a  rectangle)  (Novak,  1995).  This 
step  is  crucial  to  constructing  an  integrated  representation  of  the  common  referent  of  the 
text  and  diagram  in  memory  as  opposed  to  separate  surface-level  representations  of  the 
text  and  diagram. 

D.  Determining  the  Causal  Chain  of  Events.  When  asked  to  predict  the  behavior  of 
machines  from  static  diagrams,  people  tend  to  reason  about  machine  operation  along  the 
direction  of  causal  propagation  in  the  machine  (Hegarty,  1992;  Narayanan,  Suwa  & 
Motoda,  1994;  1995).  Therefore,  we  hypothesize  a  fourth  stage  of  comprehension  that 
involves  identifying  the  potential  causal  chains  of  events  in  the  operation  of  the  machine, 
or  "lines  of  action"  in  the  machine. 


E.  Constructing  a  Dynamic  Mental  Model  by  Mental  Simulation  and  Rule-based _ 

Inference.  The  final  stage  of  comprehension  is  that  of  constructing  a  dynamic  mental 
model  of  the  machine,  i.e.,  a  representation  of  how  the  components  move  and  constrain 
each  other’s  motion  when  the  machine  is  in  operation.  Our  previous  research  (Hegarty, 
1992;  Narayanan,  Suwa  &  Motoda,  1994;  1995)  suggests  that  people  can  often  infer  this 
information  from  a  static  diagram  by  a  process  called  mental  animation.  Computational 
models  and  empirical  evidence  suggest  that  this  is  an  incremental  process  in  which  the 
reasoner  considers  the  components  or  subsystems  individually,  assesses  the  influences 
acting  on  each,  infers  the  resulting  behavior  of  each,  and  then  proceeds  to  consider  how 
this  behavior  affects  the  next  component  or  subsystem  in  the  causal  chain.  It  depends  on 
both  prior  knowledge  (e.g.,  rules  that  govern  the  behavior  of  the  system  in  question)  and 
spatial  visualization  processes. 

6.2  Basic  Research  on  Comprehension  and  Inference  from  Complex  Visualizations 
in  the  domain  of  Mechanics. 

One  of  the  goals  of  this  grant  was  to  test  and  develop  this  model  by  conducting  empirical 
studies  of  how  experts  and  novices  comprehend  and  make  inferences  from  complex 
visualizations.  These  studies  involve  collection  and  analysis  of  several  different  types  of 
data,  including  problem  solving  accuracy,  response  times,  eye-fixation  protocols,  and 
verbal  protocols.  They  were  conducted  in  the  domains  of  mechanics  and  meteorology. 

Research  in  the  Domain  of  Mechanics.  Our  model  of  machine  comprehension 
suggests  that  an  important  component  of  machine  comprehension  is  that  of  understanding 
the  causal  chain  of  events  in  the  operation  of  the  machine.  This  can  be  a  source  of 
comprehension  failure,  especially  in  complex  machines  in  which  several  components 
move  at  once  and  there  are  more  than  one  causal  chain  of  events  or  “lines  of  action  in  the 
machine.  On  prediction  of  our  model  therefore  is  that  people  will  have  superior 
comprehension  of  mechanical  systems  if  the  components  of  the  device  are  highlighted  in 
order  of  the  causal  chain  of  events  in  the  operation  of  the  system. 

Four  experiments  at  Auburn  University  used  eye-fixation  analyses  to  examine 
mental  animation  of  static  displays  of  complex  causal  mechanisms  with  branching  and  - 
merging  causal  chains.  The  first  experiment  replicated  and  extended  our  previous 
research  on  mental  animation,  showing  that  successful  students  follow  the  causal  chain  of 
events  in  a  machine  when  mentally  animating  a  mechanical  system.  In  the  next 
experiment  we  collected  baseline  data  on  accuracy  and  response  time  of  subjects  making 
predictions  from  diagrams  of  complex  causal  mechanisms  with  branching  and  merging 
causal  chains.  The  third  experiment  examined  whether  highlighting  the  components  of 
the  device  in  order  of  this  causal  chain  of  events  affects  performance.  Highlighting 
improved  performance  on  the  problems  (the  group  who  received  highlighting  had  an 
accuracy  rate  of  60%  compared  to  40%  for  the  control  group)  but  also  lead  to  more  time 
on  task.  In  the  fourth  experiment,  the  effects  of  highlighting  were  replicated  with  less 
expert  participants 

Another  prediction  of  our  model  is  that  because  people  can  mentally  animate 
mechanical  systems  to  some  extent,  they  will  learn  more  from  both  multimedia 


presentations  if  they  attempt  to  mentally  animate  static  diagrams  of  mechanical.systems 
before  viewing  animations  of  how  these  machines  work,  or  learning  from  printed  (text 
and  diagram)  materials  explaining  how  these  machines  work.  Three  experiments  at 
University  of  California,  Santa  Barbara  investigated  the  effects  of  mental  animation  on 
learning  from  multimedia  and  printed  presentations  that  explain  how  machines  work. 
These  experiments  showed  that  people  leam  more  from  multimedia  presentations  if  they 
first  try  to  mentally  animate  the  machines  in  question.  We  conducted  extensive  analyses 
on  the  results  of  three  experiments  on  the  effects  of  mental  animation  on  learning  from 
multimedia,  including  precise  characterizations  of  the  mental  models  of  individuals  who 
learned  from  the  different  media.  A  paper  summarizing  this  work  was  accepted  for 
publication  in  Cognition  and  Instruction  (Hegarty,  Kriz  &  Cate,  in  press). 

In  collaboration  with  researchers  at  the  Applied  Physics  Laboratory,  University  of 
Washington,  we  conducted  an  additional  experiment  that  investigated  whether  people 
leam  more  from  multimedia  presentations  explaining  how  machines  work  if  the 
presentations  show  realistic,  3d  diagrams  of  the  mechanical  systems  rather  than  2D  cross- 
sectional  views.  We  had  two  professionally  authored  multimedia  presentations  built  for 
testing.  These  manuals,  one  using  2D  and  the  other  using  3D  graphics,  explain  a 
relatively  complex  but  familiar  device:  the  flushing  cistern.  These  manuals  incorporate  all 
the  design  guidelines  developed  in  our  research  so  far.  In  an  experimental  evaluation  of 
these  manuals  at  UCSB,  we  varied  (1)  whether  students  learned  from  the  hypermedia 
manuals  or  merely  from  a  labeled  diagram  of  the  machine  and  (2)  whether  they  received 
2-d  or  3-d  diagrams  in  their  training.  Results  indicated  that  those  who  received  the 
hypermedia  visualizations  had  superior  comprehension  of  the  machine,  but  there  was  no 
difference  between  those  who  received  2-d  and  3-d  views. 

The  grant  also  provided  partial  funding  for  several  other  related  basic-research 
projects  that  examined  the  roles  of  spatial  abilities  and  working  memory  in  mechanical 
reasoning.  In  one  project  (Hegarty,  2000)  I  examined  the  working-memory  demands 
involved  in  mental  animation  of  simple  machines,  using  the  3CAPS  production  system 
architecture.  This  research  suggested  that  both  spatial  and  executive  working  memory 
systems  are  involved  in  mental  animation.  Another  project  (Miyake,  Rettinger, 

Friedman,  Shah  &  Hegarty,  2001)  established  the  relation  between  these  working 
memory  systems  and  complex  spatial  visualization  abilities,  which  are  known  to  be 
involved  in  mechanical  reasoning. 

Other  research  projects  examined  the  relation  of  spatial  abilities  to  various  forms  of 
physics  problem  solving,  including  mental  animation  (Kozhevnikov,  Hegarty  &  Mayer, 
2002a;  2002b).  This  research  indicated  the  necessity  to  distinguish  between  spatial 
abilities  involved  in  imagining  manipulations  of  objects  and  spatial  abilities  involved  in 
imagining  different  perspectives  in  space  (Kozhevnikov  &  Hegarty,  2001a).  Finally,  we 
conducted  a  literature  review  in  which  we  compared  and  contrasted  research  on  explicit 
knowledge  about  mechanical  phenomena,  as  revealed  by  naive  physics  studies,  and 
implicit  knowledge,  as  revealed  by  studies  of  the  perception  of  moving  stimuli 
(representational  momentum)  (Kozhevnikov  &  Hegarty,  2001b). 


6.3  Design  principles  for  interactive  visualizations,  based  on  the  model  and 
empirical  studies  described  above. 

The  comprehension  model  and  empirical  research  described  above  suggested  the 
following  general  design  guidelines  for  interactive  visualizations  that  support  human 
performance, 

(1)  The  decomposition  guideline :  provide  cues  in  verbal  and  visual  representations 
that  help  users  decompose  the  system  or  process  being  explained  into  simpler 
components. 

(2)  The  prior-knowledge  guideline :  use  words  and  pictures  that  help  users  invoke  and 
connect  their  prior  knowledge  to  the  external  representations. 

(3)  The  co-reference  guideline :  use  interactive  and  deictic  devices  to  highlight  the 
common  referent  when  multiple  verbal  and  visual  references  in  different  media 
refer  to  the  same  object  or  component. 

(4)  The  lines-of-action  guideline:  use  words  and  pictures  that  help  a  user  understand 
the  physical,  causal  and  logical  connections  among  parts  that  determine  how  the 
behavior  of  each  part  of  the  system  or  process  influences  that  of  others. 

(5)  The  mental  simulation  guideline :  use  graphics  and  interactivity  to  encourage  users 
to  predict,  or  mentally  simulate,  the  process  or  system  that  is  being  explained 
before  providing  an  external  animation. 

(6)  The  basic  principles  guideline:  When  the  operation  of  a  system  depends  on  basic 
principles  that  might  not  be  understood  by  all  users,  describe  these  principles 
explicitly  in  the  context  of  the  system  being  explained. 

6.4  Development  and  testing  of  prototype  visualizations  based  on  the  design 
principles. 

Following  the  design  guidelines  outlined  above,  we  developed  two  different 
hypermedia  presentations,  one  in  the  domain  of  mechanics  and  the  other  in  the  domain  of 
computer  algorithms  (the  latter  presentation  was  developed  in  part  with  funding  from  the 
National  Science  Foundation  to  Dr.  Narayanan).  Details  of  the  structure  of  these  manuals 
are  described  by  Narayanan  &  Hegarty,  2002). 

We  conducted  several  experiments  comparing  these  hypermedia  to  (1)  informationally 
equivalent  printed  manuals  that  also  conformed  to  these  design  guidelines  (3)  animations 
typical  of  extant  research  and  commercial  CD-ROMS,  and  (4)  mixed-mode  explanations 
extracted  from  commercial  books.  First,  we  found  that  manuals  designed  according  to  our 
guidelines  are  more  effective  than  manuals  that  do  not  conform  to  these  guidelines  (both 
commercial  and  research  products)  (Narayanan  &  Hegarty,  2002;  Hegarty,  Narayanan  & 
Freitas,  2002).  Second,  we  found  different  results  across  domains  with  respect  to  the 
question  of  whether  computer-based  dynamic  hypermedia  presentations  are  more 
effective  than  informationally  equivalent  static  paper-based  presentations.  In  the  physical 
domain  of  mechanics,  computer-based  dynamic  presentations  were  not  superior  to 
printed  presentations.  In  the  abstract  domain  of  algorithms,  computer-based  dynamic 
presentations  were  superior  (Narayanan  &  Hegarty,  2001;  Hampson,  Narayanan  & 
Hegarty,  2002). 


6.5  Basic  Research  on  Comprehension  and  Inference  from  Complex  Visualizations 
in  the  Domain  of  Meteorology. 

One  of  the  important  goals  of  the  research  funded  by  this  grant  was  to  start  to 
generalize  our  model  of  comprehension  of  complex  visualizations  in  the  domain  of 
mechanics  to  the  new  of  meteorology.  There  are  several  differences  between  these  two 
domains.  First,  visualizations  in  the  domain  of  meteorology  are  much  more  complex  than 
in  the  domain  of  mechanics.  For  example,  typical  displays  in  this  domain  (weather  maps) 
often  superimpose  as  many  as  7  or  8  different  variables  on  the  same  map.  Second,  the 
displays  are  more  abstract  -  although  diagrams  of  machines  show  physical  (visible) 
objects,  weather  maps  often  use  visual  variables  (color,  lines  etc.)  to  represent  non-visible 
properties  such  as  temperature  and  pressure.  Third,  inferences  from  weather  maps  depend 
more  on  knowledge  of  the  domain  (compared  to  mechanical  inferences  which  can  be 
made  by  general  mental  animation  processes)  so  that  a  key  question  in  this  domain  is 
how  people  apply  their  domain  knowledge  of  meteorological  principles  to  the 
interpretation  of  weather  maps.  Therefore  our  experiments  in  this  domain  have  focused 
on  the  comprehension  and  inference  processes  of  people  with  different  amounts  of 
domain  knowledge  of  meteorology. 

In  our  initial  3  experiments  at  UCSB  we  examined  naive  students’  comprehension  of 
weather  maps  from  US  newspapers.  Naive  students  are  those  who  have  received  no 
formal  education  in  meteorology.  In  these  experiments,  we  manipulated  the  number  of 
variables  (pressure,  temperature,  precipitation)  shown  on  the  maps  and  assessed  how  this 
affected  students  descriptions  of  the  weather  and  their  verification  of  statements  about  the 
weather  in  different  locations.  Students’  descriptions  were  very  literal  descriptions  of  the 
specific  variables  shown  on  the  maps.  They  rarely  related  two  or  more  variables  or  made 
any  inferences  from  the  information  displayed.  They  were  slower  to  verify  a  fact  about  a 
particular  variable  when  more  variables  were  shown  on  the  map. 

In  further  experiments,  we  examined  people’s  ability  to  make  inferences  (weather 
forecasts)  about  changes  in  temperature  from  maps  showing  pressure  and  temperature 
patterns  across  the  United  States.  In  these  experiments  we  compared  performance  of — 
experts  (meteorologists  at  a  Naval  Research  Laboratory),  novices  (geography  students 
who  had  taken  one  course  in  meteorology)  and  naive  students  (psychology  students  with 
no  formal  training  in  meteorology). 

The  experts  were  extremely  consistent  in  their  predictions,  indicating  that  there  is  an 
agreed-upon  correct  prediction  for  each  task.  Novices  and  naive  individuals  were 
compared  on  their  ability  to  make  these  predictions  and  their  knowledge  of  the 
underlying  meteorology  principles  on  which  the  predictions  are  based.  Although  almost 
all  novices  were  able  to  state  the  principles  correctly,  and  were  much  superior  to  naive 
individuals  on  this  measure,  the  two  groups  did  not  differ  on  their  ability  to  make 
inferences  from  the  weather  maps.  Overall  performance  on  the  prediction  task  was  just  a 
little  above  chance. 


Based  on  verbal  protocols  and  patterns  of  responses  across  different  problems  we 
identified  the  specific  causal  principles  used  by  each  student  in  making  his/her 
predictions.  Results  indicated  that  whereas  students  can  articulate  individual  causal 
principles  of  weather  (e.g.  air  moves  from  areas  of  high  pressure  to  areas  of  low 
pressure),  they  have  considerable  difficulty  in  integrating  and  applying  multiple  causal 
principles  to  accurately  predict  weather  events,  A  further  experiment  showed  that  their 
failure  to  make  inferences  from  the  maps  was  not  due  to  a  failure  to  activate  the  relevant 
knowledge  (reminding  students  of  the  relevant  knowledge  did  not  affect  their 
performance). 

In  the  two  further  experiments  we  developed  multimedia  instruction  to  teach 
meteorology  principles  to  naive  students  and  varied  whether  or  not  the  principles  were 
explained  in  the  context  of  a  weather  forecasting  problem.  In  these  experiments,  we  were 
successful  in  teaching  the  principles  to  naive  students,  but  unsuccessful  in  teaching  these 
students  to  apply  the  relevant  knowledge  to  making  inferences  from  weather  maps.  These 
studies  suggest  that  knowledge  of  relevant  causal  principles  and  ability  to  comprehend  a 
weather  map  are  not  sufficient  conditions  for  being  able  to  make  inferences  from  weather 
maps. 

Finally,  we  designed  and  implemented  a  web-based  interactive  weather  map  display 
program  in  Javascript  that  allows  a  user  to  selectively  overlay  or  hide  one  of  several 
meteorological  variables  such  as  pressure,  temperature  and  dew  point.  The  program  logs 
all  user  interactions.  A  companion  program  was  developed  that  could  parse  these  logs 
and  replay  the  interactions  of  specific  users  so  that  the  experimenter  could  get  a 
qualitative  understanding  of  the  actions  of  each  user.  This  system  was  used  in  an 
experiment  at  UCSB  to  elucidate  novice  strategies  for  presentation  of  meteorological 
variables  when  reasoning  about  weather  phenomena  involving  multiple  variables. 
Analysis  of  data  from  this  experiment  is  in  progress.  Our  goal  is  to  produce  a  model  of 
novice  strategies. 

6.6  Development  of  Eye-Tracking  Laboratories 

We  set  up  eye-tracking  laboratories  with  two  trackers  from  SMI  Inc,  at  both  UCSB  and 
Auburn  University.  This  equipment  was  purchased  with  funds  from  a  related  DURIP 
grant.  This  included  several  technical  developments  as  follows: 

1.  We  developed  software  for  presenting  stimuli  and  collecting  eye-fixation  data  in 
experimental  studies 

2.  Software  for  visualizing  eye-fixation  data,  i.e.  replaying  the  stimulus  that  a 
participant  in  an  experiment  had  viewed,  while  overlaying  his  or  her  eye  fixations 
on  that  stimulus 

3.  Software  for  analyzing  eye-fixation  data,  i.e.  defining  “regions  of  interest”  in  a 
visual  display  and  examining  the  frequency,  order  and  duration  of  fixations  on 
those  regions  of  interest. 


t 


This  software  was  used  in  4  experiments  at  Auburn  and  2  experiments  at  UCSB  to  date. 
Several  experiments  have  been  concerned  with  the  effects  of  attentional  cues  in  diagrams 
and  animations  on  directing  observers’  eye  fixations  to  the  most  relevant  areas  of  a  visual 
display  and  on  the  effects  of  these  attentional  cues  on  problem  solving  and  learning  from 
multimedia  displays.  One  other  experiment  examined  the  relation  between  students’  eye 
fixations  and  their  verbalizations  when  they  solved  mechanical  troubleshooting  problems 
and  gave  “think  aloud”  protocols. 

We  also  accomplished  some  further  development  of  a  program  called  the  Restricted 
Focus  Viewer  (RFV),  developed  at  Monash  University  in  Australia.  It  follows  a  user’s 
mouse  movements  and  only  reveals  parts  of  a  stimulus  image  around  the  cursor  location 
while  hiding  the  rest  of  the  image.  It  is  a  relatively  inexpensive  alternative  to  using  an 
actual  eye  tracker,  and  does  not  suffer  from  some  of  the  limitations  one  faces  when  using 
the  eye  tracker  in  an  experiment.  It  allows  us  to  easily  carry  out  pilot  studies  in  eye 
tracking  before  running  actual  eye  tracker  experiments.  This  was  accomplished  by 
inviting  the  author  of  the  RFV,  a  Computer  Science  student  at  Monash  University,  to  visit 
Dr.  Hegarty's  lab  for  a  month  in  summer.  This  software  was  used  in  two  experiments  at 
UCSB  to  examine  information-seeking  patterns  of  novices  when  they  scanned  weather 
maps  from  US  newspapers  in  order  to  verify  a  specific  fact  about  the  weather  in  different 
locations 

Finally,  an  interactive  data  analysis  and  visualization  program,  called  RFVDAT,  was 
developed  at  Auburn  to  parse  data  from  the  RFV.  This  program  allows  the  experimenter 
to  select  any  regions  of  interest  on  the  stimulus  and  receive  data  how  long  the  subject 
spent  viewing  these  regions  of  interest,  the  order  in  which  they  viewed  these  regions  etc, 

6.7  Dissemination: 

Our  research  was  disseminated  through  1 1  journal  articles,  3  book  chapters,  and  6  peer- 
reviewed  conference  presentations.  We  also  made  several  invited  presentations  on  our 
work  as  listed  below 

7,  Best  Accomplishments 

•  The  development  of  a  cognitive  model  of  multimodal  comprehension. 

•  The  generation  and  dissemination  of  guidelines  for  the  design  of  multimedia 
information  presentations. 

•  Design  and  evaluation  of  several  versions  of  multimedia  manuals  that  conform  to 
the  cognitive  model  and  design  guidelines. 

•  The  experimental  demonstration  that  manuals  (computer-based  or  printed) 
designed  according  to  our  guidelines  are  more  effective  than  manuals  that  do  not 
conform  to  these  guidelines  (both  commercial  and  research  products)  in  the  two 
domains  of  machines  and  computer  algorithms  (Narayanan  &  Hegarty,  2002). 


•  Experimental  demonstration  that  when  a  text-and-diagram  presentation  is 
designed  according  to  principles  that  we  have  so  far  developed  in  our  research,  it 
does  not  have  to  be  delivered  on  a  computer  (i.e.  it  can  be  paper-based)  in  order  to 
effectively  support  the  comprehension  and  inference  tasks  in  the  mechanical 
domain  (Narayanan  &  Hegarty,  2002;  Hegarty,  Narayanan  &  Freitas,  2002; 
Hegarty,  Kriz  &  Cate  in  press). 

•  Experimental  demonstration  that  when  a  text-and-diagram  presentation  in  the 
mechanical  domain  is  designed  according  to  principles  that  we  have  so  far 
developed  in  our  research,  it  does  not  matter  whether  a  2-d  or  3-d  diagram  is 
shown. 

•  Demonstration  that  attentional  cues  in  diagrams  (that  guide  attention  to 
information  as  it  is  needed  for  problem  solving)  can  enhance  human  problem 
solving  performance. 

•  Demonstration  that  people  learn  more  from  both  dynamic  computer-based 
animations  and  static  text  and  diagrams  describing  how  machines  work  if  they 
first  mentally  animate  static  diagrams  of  these  machines  (Hegarty,  Kriz  &  Cate,  in 
press). 

•  Experimental  demonstration  that  although  novices  can  articulate  individual  causal 
principles  in  the  domain  of  meteorology,  they  have  considerable  difficulty  in 
integrating  and  applying  multiple  causal  principles  to  accurately  predict  a  weather 
event.  Therefore,  (1)  ability  to  understand  a  visualization,  and  (2)  knowledge  of 
relevant  causal  principles  are  not  sufficient  conditions  for  making  causal 
inferences  from  weather  maps.  Thus,  although  novices  can  articulate  individual 
causal  principles  in  the  domain  of  meteorology,  they  have  considerable  difficulty 
in  integrating  and  applying  multiple  causal  principles  to  accurately  predict  a 
weather  event, 

•  Experimental  demonstration  that  naive  students’  descriptions  and  interpretations 
of  weather  maps  are  influenced  by  the  number  of  variables  shown  on  the  maps. 

8.  Impact/Applications 

I.  Impact  of  Basic  Research 

Our  experimental  results  have  contributed  to  the  further  development  of  a  theory  of 
multimedia  instruction.  We  have  proposed  new  design  guidelines  that  exploit  the  power 
of  inference  and  mental  animation  in  learning  from  multimedia.  The  empirical  result  that 
our  hypermedia  manuals,  whether  on  paper  or  on  computer,  are  more  effective  than 
commercially  available  materials  indicate  that  our  guidelines  offer  a  significant 
improvement  over  the  conventional  wisdom  of  designers  of  multimedia  software.  These 
results  question  widely-held  beliefs  about  multimedia,  and  as  a  result  of  dissemination 


will  influence  multimedia  instructional  design  practice  in  the  commercial  and  military 
sectors. 

Our  experimental  results  in  the  domain  of  meteorology  have  pointed  to  limitations  in  the 
comprehension  of  weather  maps  by  naive  and  novice  students.  These  results  suggest  that 
instruction  in  meteorology  needs  to  be  improved  so  that  causal  principles  are  introduced 
in  the  context  of  practical  tasks  such  as  weather  forecasting.  In  particular,  students  need 
to  be  taught  how  to  weigh  and  integrate  different  causal  principles  in  weather  prediction. 

II.  Impact  of  Technology  Development 

The  eye  tracking  laboratories  at  Auburn  and  Santa  Barbara  are  two  of  the  very  few 
research  groups  in  the  United  States  investigating  the  application  of  eye  tracking  to  HCI 
and  visualization  research.  We  have  developed  various  software  programs  for  facilitating 
the  analysis  of  eye  movements  easier,  and  for  realizing  gaze-contingent  displays.  These 
technologies  have  mainly  been  used  in-house  for  supporting  basic  research.  Since  the 
focus  of  this  research  program  was  not  developing  and  disseminating  technology  per  se, 
the  impact  of  the  technology  we  have  developed  is  indirect  in  terms  of  its  support  of  basic 
research. 

9.  Results 

Contrary  to  conventional  belief,  we  found  no  evidence  that  students  leam  more  from 
animations  than  from  static  diagrams  of  mechanical  systems.  We  propose  that  this  is 
because  people  can  mentally  animate  static  diagrams.  Conditions  that  induce  them  to 
mentally  animate  static  diagrams  include  question  answering,  reading  text  that  describes 
how  the  parts  of  a  machine  move  and  viewing  several  diagrams  (small  multiples) 
showing  different  phases  in  the  operation  of  the  machine. 

Contrary  to  conventional  belief  among  multimedia  designers,  we  found  no  evidence  that 
students  leam  more  from  3-d  diagrams  of  mechanical  systems  compared  to  2-d  diagrams, 
that  they  are  better  at  making  predictions  of  device  behavior  from  3-d  diagrams,  or  that 
they  find  3-d  diagrams  more  interesting  or  engaging.  This  suggests  that  additional 
expense  involved  in  producing  3-dimensional  diagrams  is  not  justified,  at  least  in  the 
situations  we  have  studied  to  date. 

Visualizations  that  guide  a  problem  solver's  attention  along  causal  pathways  in  a  complex 
domain  enhance  attention  to  the  relevant  components  of  the  problem  and  accuracy  in 
problem  solving,  at  the  cost  of  increased  response  time. 

Naive  students  take  longer  to  verify  facts  from  more  complex  weather  maps  (showing 
more  variables)  compared  to  simpler  weather  maps.  This  suggests  that  when 
communicating  to  naive  individuals,  visual  displays  should  show  only  the  relevant 
information  and  no  additional  information. 


After  taking  a  course  in  meteorology,  novice  students  can  articulate  important  causal 
principles  in  this  domain,  but  have  considerable  difficulty  in  integrating  and  applying 
multiple  causal  principles  to  accurately  predict  a  weather  event.  This  suggests  that 
instruction  in  meteorology  needs  to  be  improved  so  that  causal  principles  are  introduced 
in  the  context  of  practical  tasks  such  as  weather  forecasting.  In  particular,  students  need 
to  be  taught  how  to  weigh  and  integrate  different  causal  principles  in  weather  prediction. 

10.  Technology  Transfer 

Our  main  means  of  facilitating  technology  transfer  has  been  through  research 
presentations  at  forums  attended  by  scientists  and  engineers  including  those  from 
industry  and  military.  Through  various  presentations  we  have  disseminated  the  results  of 
our  research  at  cognitive  psychology,  human-computer  interaction,  multimedia  and 
interactive  system  design  forums  attended  by  scientists  and  engineers  including  those 
from  industry  and  military. 

Hegarty  made  a  presentation  at  the  workshop  on  Interactive  Meteorology  and 
Oceanography  in  August  2001,  at  the  Naval  Pacific  Meteorology  and  Oceanography 
Center,  San  Diego,  CA,  thereby  making  Navy  METOC  personnel  aware  of  results  from 
our  research.  Narayanan  presented  results  from  this  research  at  the  Navy  Center  for 
Applied  Research  in  Artificial  Intelligence,  Naval  Research  Laboratory,  in  May  2002,  at 
the  ONR  Workshop  on  Attention,  Perception  and  Data  Visualization,  George  Mason 
University,  in  May  2002  and  at  the  DARPA  Augmented  Cognition  Conference  in  Austin, 
Texas  in  December  2001. 

Hegarty  also  consulted  with  Navy  Meteorology  personnel  regarding  the  design  of  stimuli 
for  experiments  in  meteorology  and  Dr  Ted  Tsui  at  the  Naval  Research  Laboratory  in 
Monterey.  She  visited  NRL  in  November  2001  and  to  collect  data  from  expert 
meteorologists  and  to  consult  with  these  experts  on  the  design  of  future  research  studies 
in  the  domain  of  meteorology. 

This  is  a  basic  research  project  and  therefore  no  actual  products  have  been  produced. 
However,  our  research  has  generated  a  set  of  guidelines  for  designing  effective 
multimedia  presentations  which  were  communicated  to  another  ONR-supported  group  — 
Applied  Physics  Laboratory  (APL),  University  of  Washington  -  and  we  had  them  design 
two  hypermedia  manuals  through  a  subcontract.  This  mutually  beneficial  collaboration 
informs  APL  designers  of  the  results  of  our  research  and  enables  them  to  construct 
multimedia  visualizations  that  are  based  on  empirically-validated  cognitive  principles. 

11.  Statistics 

UCSB:  Graduate  Students/Postdoctoral  scholars  supported  at  least  25%  from  this  grant 
for  at  least  1  quarter: 

Graduate  Students:  Non-minority  women:  4 

Minority  man:  1 


Non-minority  man:  1 

Post  Doctoral  scholar:  Non-minority  woman:  1 
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